• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    Dron characterized in that each of the external parts is covered or surrounded by a screen of a material of uniform reflectivity and color. And it includes an image capturing device that makes other drones, equally coated, invisible in the photographs taken. This device applies a new method that allows to distinguish the zone that occupies in the image each drone, for its later elimination by techniques of thresholdizaciºn by color and of restoration of image. (Machine-translation by Google Translate, not legally binding)
    公开号:ES2585251A1
    申请号:ES201531872
    申请日:2015-12-22
    公开日:2016-10-04
    发明作者:David MIRAUT ANDRES
    申请人:Universidad Rey Juan Carlos；
    IPC主号:
    专利说明:

    INVISIBLE DRON
    SECTOR OF THE TECHNIQUE
    The present invention falls within the technical area of unmanned aerial vehicles and their methods of use. Specifically, that which concerns unmanned aerial vehicles that are used for the acquisition of images and video. BACKGROUND OF THE INVENTION
    Drones or unmanned aerial vehicles (VANT), whose acronym in English correspond to UAV (Unmanned Aircraft Vehicle), have evolved rapidly in recent years. These types of unmanned aircraft have gone from exclusively military purposes to widely used in all types of civilian tasks. The diversity of these tasks has resulted in the specialization of the designs of these aircraft as well as their control systems, with the consequent variety of shapes, sizes, configurations and features adapted to each of the tasks.
    Typically, it is usually known by the acronym UAS (Unmanned Aircraft System) to the assembly that includes the aircraft, the station or control system from where it is operated and (optionally) the unit that is responsible for data transmission.
    Drones can be considered as platforms that carry inside a set of sensors and are intended to obtain geospatial data. This data can guide your piloting, to the point that, depending on the sophistication of the control system, the drone can operate completely autonomously under the supervision of a pilot, determining in real time the path to be traveled.
    However, the most popular models are usually controlled remotely from a ground station by an operator. Therefore, the term "unmanned" may be the source of some confusion, since although it is true that no crew is traveling inside the vehicle, it is also true that there is continuous communication between the UAV and
    operators on the ground, be they pilots, controllers or any other type
    of operator related to the monitoring of the aircraft.
    The lack of regulation or its laxity in some countries, as in the case of the United States of America, has favored the popularization of this type of technology and the creation of new market niches. For example, the US FAA (Federal Aviation Administration) allows fans to use small aircraft within the operator's field of vision and at heights below 400 feet in areas that are not densely populated.
    On the other hand, the audiovisual commercial sector has shown a special interest in the use of such devices, since unmanned aircraft can be used to make video and photo plans that would otherwise require a helicopter or a manned aircraft, that supposes a strong reduction of costs.
    This technology is especially interesting in the recording of sporting events. A clear example of its application's success was the 2014 Winter Olympics in Sochi, where drones with cameras were used to film ski and snowboard events. In both cases, this type of devices allowed to take images of great spectacularity when approaching the athletes, and to provide greater flexibility than the systems of cameras placed in suspended cables.
    Given its relative low cost, an amateur can record himself during the practice of any outdoor sport if the UAS is able to follow a visual mark or a radio signal; concentrating on performing sports, without the need for an operator or pilot to control the flight path. Products that meet these characteristics can be found on the market such as AirDog [http://www.airdog.com/] of Helico Aerospace Industries US LLC, Lily [http://www.lily.camera/] of Lily Robotics, Inc. , HEXO + [https://hexoplus.com/] of Squadrone System or PlexiDrone [https://www.indiegogo.com/projects/plexidrone-making-aerial-photography-a-breeze#/] DreamQii.
    This fact can be generalized and it is easy to imagine sporting events in which several fans or even television networks simultaneously use this technology to capture and transmit live sequences of great visual richness.
    The most sophisticated UAS have mechanisms to avoid colliding with vegetation
    or any other type of obstacle in your path, including other drones even if they are not piloted in a coordinated manner.
    However, when more than one camera drone is used to record scenes on the same group of actors or athletes, it is especially difficult to obtain good frames without these recording instruments being present in the field of vision. So, in good part of the recordings made with each of the drones, in addition to recording the people of interest, they also totally or partially capture other drones, which can distract the viewer's attention.
    Since the drones in this case are auxiliary elements that help to make the recordings, many television or filmmakers discard the shots in which they are visible, since they are not interested in appearing in the composition. This situation is all the more common the higher the number of unmanned aerial vehicles with cameras in the recording area.
    This problem is almost inevitable, even using the most modern trajectory planning systems like the one recently published by Niels Joubert, Mike Roberts, Anh Truong, Floraine Berthouzoz, and Pat Hanrahan in SIGGRAPH Asia 2015, entitled An Interactive Tool for Designing Quadrotor Camera Shots . Different camera operators and autonomous systems can choose frames from angles in which other drones, or their reflection, are shown in the scene.
    This situation is inconvenient. The presence of these types of devices, to which viewers are not yet accustomed, tends to distract viewers' attention. Its elimination through postproduction tools requires the manual work of qualified professionals. Thus, the cost in time precludes its realization in live broadcasts.
    It would therefore be desirable for drones to fulfill their function without taking center stage in the photographs.
    The applicant of the present invention is unaware of the existence of a background that satisfactorily solves the problem presented.
    The present invention offers a solution to this technical problem, both in the case of live productions and in recordings that are transmitted in deferred. This solution allows to detect and distinguish the portion of the image in which the drones appear in the field of vision, so that it can later be eliminated using digital image restoration techniques (inpainting in English). Thus, the images are obtained so that the content of interest is preserved without elements that divert the
    viewer's attention, by eliminating the presence of drones that meet the
    claimed characteristics. EXPLANATION OF THE INVENTION
    Drones are tools that allow camera operators to obtain spectacular images. The sophistication of the control systems (UAS) already makes it possible for your trajectory to be calculated in an unsupervised manner and the taking of video sequences is automated at a very low cost. These factors favor the use of a large number of these devices in all kinds of outdoor events, such as in extreme sports competitions.
    As already indicated, the presence of swarms of unmanned vehicles that try to capture images of an event simultaneously, entails the problem of the visibility of these devices in the recorded sequences.
    The only way to prevent a drone from appearing in the field of view of another drone's camera is to restrict the space in which they can fly around points of interest. Which in turn makes it impossible to take simultaneous images from certain angles.
    In the case of film production, the sequences must be repeated with the consequent investment in personnel and time for their realization. But in the case of a live event, this restriction is especially inconvenient, since the scenes cannot be repeated and the limitation in the positions of the drones implies the loss of those planes.
    Instead of limiting the position of the drones, the present invention proposes to make the recordings in a natural way according to the criteria of the producer and subsequently eliminate the presence of the drones in the photographs or video sequences using digital image processing techniques.
    Image reconstruction algorithms for the removal of objects (in English inpainting) require that previously distinguish those parts of the image to be removed. Its automatic determination is still an open field in research, since - in general - the objects to be removed depend on the preferences of the user and the context offered by the composition in the photograph. However, in the case of images taken from multiple perspectives with a set of drones, the objects to be removed are usually the drones themselves that carry the cameras from which the images have been taken. Our invention facilitates that an automated image processing system can differentiate between drones and the rest of the scene in the captured image, by means of a simple color thresholding, so that it can be segmented and eliminated by inpainting techniques.
    For this, it is proposed that the drone be covered with a material with a uniform reflectivity, of a uniform and characteristic color, unlike the plastic or metallic finish that the drones marketed so far usually present. A material with uniform reflectivity causes the light incident on the surface to be reflected in all angles equally (so that materials that have this type of interaction with light are usually known as lambertians), which prevents reflections from occurring or shine on its surface.
    The coating can be done in two ways: directly on the fuselage and the wings of the drone or by placing a screen around it made of a material that meets the described properties.
    This screen panoramicly surrounds the drone, so that it is not visible from the point of view of other drones. The screen has holes to not obstruct the camera's vision and to let the air flow through it. Although the screen can affect the aerodynamics of the device, its geometry and lightness do not alter the flight capacity and it has the advantage of exercising additional protection against possible blows or forced landings. The screen allows the use of the present invention by adapting the recording equipment and drones already existing in the market.
    In this explanation both alternatives are proposed simultaneously and interchangeably, being essentially part of the same invention.
    As in any material, on the external surface of the drone there is a smooth variation in irradiance (flow of light energy per unit area), due to the angle of incidence of light and the curvature of its surface, which results in a shadow on the surface of the fuselage and the screen. For the thresholding process to identify the area of the image corresponding to the drone by selecting a set of colors close to the chosen one, it is convenient that the segmentation be done in terms of the chromaticity of said color in colorimetric spaces such as CIE 1976 L * a * b or the CIE 1976 L * u * v, in which the intensity of the light is specified with an independent component (which can be ignored to increase robustness against variation
    soft irradiance). This strategy is similar to the one applied in the choma-key in the
    audiovisual industry
    If any element of the scene has the same color as the drone coating, both would be removed from the image. This unwanted effect is the main limitation of thresholding and segmentation by color. Therefore, although it would be feasible to cover the drone of any uniform matte color, it is preferable that the color is difficult to find in the scenarios in which the video shots are to be taken.
    Since, regardless of race, the color of human skin pigmentation is eminently red, it is preferable that the material covering the drone has a uniform color as far as possible from red in the aforementioned colorimetric spaces. As, for example, certain shades of green or blue that are not usually found in natural or urban environments. An example of this type of colors is the light green that is formed by taking only the green channel and a high level of brightness in the RGB colorimetric space with normalized coordinates [0,1,0], which stands out for "artificiality" as it is not common in the mentioned environments.
    Many of the unmanned aerial vehicles used for video recording resemble helicopters with multiple rotors for their support and propulsion in the air. These types of designs allow them to use smaller blades, which have several advantages: they provide the possibility of raising the flight with less kinetic energy, that the pilot has greater control in the maneuvers and, more importantly, reduces the possible damage in the case of a collision with a person. In order to protect people and the vehicle if both collide, the blades are usually surrounded by fuselage bars that, as a shield, avoid part of the possible blows without reducing the volume of air that must move. These structures must be removed from the images, as well as the possible bias in the color produced by the blade on the stage objects that are seen through it when it is in motion.
    Another aspect to keep in mind is that the lens cannot be colored, because the paint would block the passage of light and images could not be captured. But it must be removed from the image just like the rest of the drone that appears in the field of view.
    Therefore, since the fuselage, the wings and the screen usually surround both the recesses of the blades and the objective when viewed in the images taken from another vehicle, the proposed system detects and marks not only those areas that have the chosen color
    but also those that are completely surrounded by areas that have the color
    chosen, or that are framed by these areas and the border of the field of vision.
    Thus, a set of drones whose pieces are covered by a uniformly colored Lambertian material can be used for the proposed purpose, in combination with digital image capture devices in which the method comprising the following steps is carried out:
    one. Capturing the image with a digital camera
    2. The generation of masks as a result of the process of thresholding the image with respect to the uniform color of the material that covers the drones, in which those pixels whose color coincides or has a small variance in chrominance with respect to the chosen color are individually marked.
    3. The grouping and labeling of the pixels marked in adjacent regions, for easier identification.
    Four. The determination of the contours of the labeled regions.
    5. In order to also mark the objective lens and the cavities where the drone blades are located, it is calculated whether the same region is bounded by several contours and if they contain each other. In which case the one with the largest area (corresponding to the outer contour) is chosen and all the pixels it contains are labeled as belonging to the main mask in that area.
    6. The search for the labeled regions of the mask in the upper, lower, right and left limits of the image.
    7. The selection of labeled regions located on the edges of the image.
    8. Step 5 is repeated in the previously selected regions, considering that the border of the image acts by delimiting contours if the length necessary for this is less than a certain threshold (such as one sixth of the length of the smaller side of the image) .
    9. The application of an image restoration algorithm for the removal of objects (inpainting) on the selected portions of the mask through the labeled regions, so that the portions of selected masks are filled with information from the rest of the image (or from other images). There are numerous known algorithms that allow you to replace portions of the image with the information present in
    the rest of the image, such as the techniques protected by US patents US 6987520, US 7551181, US 7755645 and US 7840086. At this stage any of the mentioned techniques or other more sophisticated techniques that take advantage of temporal coherence in the sequences could be used. of video, on the areas marked with the masks.
    At the end of the stages, no other drones will appear in the resulting image. Since instead a set of patterns will be drawn that replace them visually. The better the image restoration algorithm chosen, the more realistic the result of such substitution, which makes each of the drones disappear. BRIEF DESCRIPTION OF THE DRAWINGS
    To complement the description that is being made and in order to help a better understanding of the characteristics of the invention, a set of drawings is attached as an integral part of said description, where illustrative and non-limiting nature has been represented. next:
    Figure 1.- Shows an artistic representation of a photograph taken from a drone in a sporting event in which a runner and other drones covered in a lambertian material of uniform color according to the claims appear in the frame. In a dashed line the field of vision of each of the image capture devices is symbolized, being exposed to each other in the shots of the race and hiding part of the scene of interest.
    Figure 2.- It shows the pixels marked after the thresholding by color in the artistic representation of the photograph of figure 1. The holes left by the lenses (5) and the spaces in which the blades (6) of the drones
    Figure 3.- Shows the masks (7) generated in the artistic representation of the photograph of figure 1, after the application of step 5 of the described method.
    Figure 4.- Shows the result of eliminating the areas covered by the masks (7) in the artistic representation of the photograph of figure 1, before carrying out step 9 of the described method. Without the application of image restoration techniques the result can be as disturbing to the viewer as the presence of the drones, by distracting their attention from the sporting event.
    Figure 5.- Shows an artistic representation of the result of applying the proposed method completely to a photograph when the drones are covered by a lambertian material of uniform color.
    Figure 6.- Shows a perspective view of a drone that has been incorporated into a screen that surrounds it panoramicly, of Lambertian material of uniform color with an aperture (12) for capturing images with the objective. PREFERRED EMBODIMENT OF THE INVENTION
    A perspective view of a particular embodiment of the invention is shown in Figure 6. In this case the drone (11), quadcopter type, has been covered panoramicly by a screen (8) that has a recess (12) that surrounds the slow (5) of the lens (10), which allows the taking of images without the screen blocking the field of view. No additional openings have been necessary, since the screen does not impede the passage of air for the operation of the blades. The screen is fixed to the quadcopter by means of a series of fasteners (9) in the structure that protects the propellers.
    Alternatively, a similar result could have been obtained if all the outer parts of the quadcopter were covered with a reflective material and uniform color.
    In this example, we chose to use a light green matte paint on the screen (8) (due to the limitations of color reproduction in the figures, it is shown in a gray tone). On the one hand, this color has been chosen because it is markedly different from the tone of human skin, and on the other hand because it is less common than blue in urban environments (the sky usually has bluish tones and blue clothes, such as jeans , is widely used in the West). In addition, digital camera sensors tend to be more sensitive to green, because the Bayer pattern that filters have on their surface dedicates more sensors to the green channel (mimicking the greater sensitivity of the human visual system to this frequency range in the visible spectrum), so less light is needed to illuminate the greenish colors.
    The automated system would consist of one or several drones (1,2,3,11) whose outer surface is covered with a material of reflectivity and uniform color, in this case light green, either directly or with a screen (8). Equipped with a digital image acquisition device that in turn is composed of:
    - an optical sensor in the visible range, together with the set of lenses (5) necessary to
    focus image
    - a data processing unit (typically a digital processor)
    - one or more memories
    - a communication unit
    These last three act in a coordinated way (by means of a program that encodes the steps) on the sensor to register and store the photograph at the moment indicated by the user, and then apply the method described above.
    In this particular embodiment, the color thresholding that selects the pixels that potentially correspond to the areas occupied by the drone considers as candidates those with a chrominance variance of less than 4% with respect to the chosen color. In the example of the representation of a photograph of Figure 1, the external surface of the drones would be detected and labeled as regions of the masks. However, as shown in Figure 2, the gaps (6) in which the propellers rotate reveal partially through them and, like the lenses (5) of the lenses (10), are not labeled as part of the masks through a simple thresholding by color.
    For this reason, the proposed method identifies the external contours of each previously marked region and fills all the pixels inside, with the identifier corresponding to that of the main mask in the affected area.
    On some occasions, the hollow of a blade or a portion of a lens is located at the edge of the image, so that from the point of view of the drone that takes the image, the hole of the blade or the lens is no longer surrounded for the uniform color and reflectivity material, so that the step described above would not label these regions for later replacement in the image. In order to incorporate these elements in the labeling of the masks, it is considered that the border of the image can act as part of the outer contour of the same mask if the length necessary to connect two different parts of it is less than one sixth of the length of the side of the image (the smallest in the case of being rectangular). And the set of pixels that is covered by the contour so extended is labeled, for later treatment with the inpainting algorithm, as can be seen in Figure 3.
    Finally, the image restoration algorithm applied is that published in the article "Object Removal by Exemplar-Based Inpainting" by A. Criminisi, P. Perez and K. Toyama. Algorithm that has been mentioned above, since it is protected with US patent 6987520 with the title "Image region filling by exemplar-based inpainting".
    5 Figure 5 shows the image that would be obtained as a result of the application of the method by means of the automated system proposed in this invention with invisible drones.
    Once the nature of the present invention has been sufficiently described, as well as a preferred embodiment, it only remains to be added that said invention may suffer
    10 certain variations in form, materials, parameters and applicable image restoration algorithms, as long as said alterations do not substantially vary the characteristics claimed below.
    权利要求:
    Claims (4)
    [1]
    one. Method for the elimination by a drone of the images of other drones present in the images captured by it, characterized in that the method comprises the following steps:
    [2]
    2. Drone, for the implementation of the method of elimination in an image according to the preceding claim, characterized by being coated with a uniform reflectivity and color material, and being provided with a digital image acquisition device in which drones are eliminated by a color similarity criterion that includes:
    to. image capture with a digital camera;
    b. the generation of masks due to image thresholding with respect to
    a uniform color of the material that covers or surrounds the drone, in which they are marked
    individually those pixels whose color has avarianceinchrominance
    less than 4% with respect to the chosen color;
    C. the grouping and labeling of the pixels marked in adjacent regions;
    d. the determination of the external contours of the labeled regions;
    and. thegroupYlabelledfromthepixelsinheinsidefromthecontours
    delimited;
    F. the search for the tagged regions of the mask in the upper limits,
    bottom, right and left of the image;
    g. the selection of mask portions located at the edges of the image;
    h. the determination of new external contours of the same selected region
    when different portions of it are connected, when considering the border of the
    image as part of said contour, provided the length of the extension is
    less than one sixth of the smaller side of the image;
    i. the grouping and labeling of the pixels inside the new contours
    delimited;
    j. the application ofainpainting algorithm on the mask portions
    labeled in the image.
    - an image capture system (2);
    - a processor; -one or several memories in communication with the processor and the image acquisition system, which store an application executable by the processor; wherein the application can, in execution, perform the method according to claim 1.
    [3]
    3. Drone, according to the preceding claim, characterized in that the coating of
    5 uniform reflectivity and color of each of its external parts, except the lens of the capture device, is applied directly to them.
    [4]
    4. Drone according to the preceding claim, characterized in that the uniform reflectivity and color coating is carried out by means of a screen (8) surrounding the total drone
    or partially and has at least one opening (12).
    类似技术:
    公开号 | 公开日 | 专利标题
    JP2021151551A|2021-09-30|System to assist user to determine distance between user and landmark
    US10126554B2|2018-11-13|Visual perception enhancement of displayed color symbology
    US9910297B1|2018-03-06|Eyewear with chroma enhancement
    Naseer et al.2013|Followme: Person following and gesture recognition with a quadrocopter
    US8054312B2|2011-11-08|Material for motion capture costumes and props
    US20100245387A1|2010-09-30|Systems and methods for combining virtual and real-time physical environments
    US20130258044A1|2013-10-03|Multi-lens camera
    CN107534764A|2018-01-02|Strengthen the system and method for image resolution ratio
    US10495884B2|2019-12-03|Visual perception enhancement of displayed color symbology
    ES2585251B2|2017-01-26|INVISIBLE DRON
    US20180302572A1|2018-10-18|Three-dimensional image capturing system and method for obtaining three-dimensional images
    Smith2016|The photographer's guide to drones
    Prasad et al.2015|Quadcopter-based stagnant water identification
    CN112154654A|2020-12-29|Match shooting method, electronic equipment, unmanned aerial vehicle and storage medium
    Northrup2014|Tony Northrup's DSLR Book: How to Create Stunning Digital Photography
    CN109074199A|2018-12-21|For detecting the method and system of barrier at the projected position in image
    CN106023215A|2016-10-12|Method and system for distinguishing field crops from background
    Ang2013|Digital Photography Masterclass
    ES2551390B2|2016-03-30|INVISIBLE MONOPIE
    US9948914B1|2018-04-17|Orthoscopic fusion platform
    CN108646776A|2018-10-12|A kind of imaging system and method based on unmanned plane
    Hunt Jr et al.2002|Comparison of remote sensing imagery for nitrogen management
    JP6690106B1|2020-04-28|Determination device, imaging system, and moving body
    Hedgecoe2006|The art of digital photography
    US20210080228A1|2021-03-18|Camouflage patterns, arrangements and objects utilizing the same
    同族专利:
    公开号 | 公开日
    ES2585251B2|2017-01-26|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    EP1510969A1|1996-12-20|2005-03-02|Canon Kabushiki Kaisha|Subject image extraction method and apparatus|
    US20110164138A1|2010-01-04|2011-07-07|Toussaint Taliaferro Tyson|Process for the Automatic Conversion of Film Images into Programmable OOP Objects for Movie|
    US20140169667A1|2012-12-19|2014-06-19|Qualcomm Incorporated|Removing an object from an image|
    法律状态:
    2017-01-26| FG2A| Definitive protection|Ref document number: 2585251 Country of ref document: ES Kind code of ref document: B2 Effective date: 20170126 |
    优先权:
    申请号 | 申请日 | 专利标题
    ES201531872A|ES2585251B2|2015-12-22|2015-12-22|INVISIBLE DRON|ES201531872A| ES2585251B2|2015-12-22|2015-12-22|INVISIBLE DRON|
    [返回顶部]